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ABSTRACT: The continuous advancement of natural user interfaces (NUIs) allows for the 
development of novel and creative ways to support collocated collaborative work in a wide range 
of areas, including teaching and learning. The use of NUIs, such as those based on interactive 
multi-touch surfaces and tangible user interfaces (TUIs), can offer unique opportunities for 
learning, for automatically capturing digital traces of students' multimodal interactions, and for 
facilitating multi-user exploration of student data. We used a composite framework to 
characterize our first-hand experiences and a small number of related deployments. The 
dimensions of analysis considered include the orchestration activities involved, the phases of 
pedagogical practice supported, the target actors, the iteration of the LA process, and the levels 
of impact of the LA deployment. Results from our analysis helped us identify the current trends, 
gaps, challenges, and pedagogical opportunities of the application of LA and the use of NUIs for 
supporting learning . 1 
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1 INTRODUCTION 

There has been a growing interest in the potential of learning analytics (LA) for supporting mobile and 
online learning activities. However, to a large extent, student learning still happens in face-to-face (f2f) 
environments (Bowers & Kumar, 2015). The development of effective f2f communication and 
collaboration skills are key 21 st century competencies for employability and lifelong learning (Lee, Tsai, 
Chai, & Koh, 2014). Blended learning strategies and massive online courses have become popular targets 
for LA solutions (Kay, Reimann, Diebold, & Kummerfeld, 2013; Picciano, 2014), but they are primarily, or 
wholly, focused on the non-f2f online part of student engagement in learning activities. This is in part 
because it is easier to capture the logs of student interactions when the activity and communication are 
mediated by computers. By contrast, logging traces of student activity in collocated f2f settings, such as 


1 An earlier, shorter version of this paper (Martinez-Maldonado et al., 2016b) is the foundation for this article, which has been 
extended in light of feedback and insights from LAK '16. 
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traditional classrooms, can impose several practical and technical challenges (Martinez-Maldonado, 
Clayphan, Ackad, & Kay, 2014). 


Collaboration demands the development of a number of skills that are important not only in educational 
settings but also to tackle real-world challenges (Scheuer, Loll, Pinkwart, & McLaren, 2010). 
Collaborative tasks typically require brainstorming, deliberation, negotiation, and argumentation, 
usually in the service of some form of artefact design that can embody the knowledge of the group 
(Stahl, 2006). These forms of collaborative work can be powerful vehicles for authentic learning and 
typically have a major f2f element (Olson, Teasley, Covi, & Olson, 2002). In short, collaboration between 
students has the potential to greatly enhance learning (Roschelle & Teasley, 1995) and LA methods can 
provide an alternative way for researchers to visualize learning processes in their full complexity 
(Siemens & Baker, 2012) — not just in online learning environments, but also in co-located settings. 

The continuous advancement of natural user interfaces (NUIs) allows for the development of novel and 
creative ways to support collocated collaborative work in a wide range of areas. A NUI is a system for 
human-computer interaction that the user operates through intuitive actions related to natural, 
everyday human behaviour and that impose fewer barriers between the user and information (Wigdor 
& Wixon, 2011). Emerging technologies such as gesture recognition, object tracking, multi-touch screens 
and tangible interfaces allow for the deployment of novel NUIs that are generating a shift in human- 
computer interaction. The proliferation of NUIs is also opening a broader range of possible applications 
to facilitate and enrich f2f activities in educational contexts (Dillenbourg & Evans, 2011). This is of high 
significance from a LA perspective, since these technologies can make visible group processes that until 
recently have remained invisible or unquantifiable. Increasingly over the last two decades, these 
technologies have been moving from research to commercial applications (Ardito, Buono, Costabile, & 
Desolda, 2015; Evans & Rick, 2014). 


In this paper, we focus on multi-user interactive surfaces and tangible interfaces — devices designed to 
enable simultaneous interaction by one or more users. Examples of these include interactive tabletops, 
interactive whiteboards (IWB), tangible interfaces, and smaller-scale devices such as tablets, which can 
allow transitions between individual and group work (Scott, Grant, & Mandryk, 2003). Interactive 
surfaces and tangibles provide more flexibility in creating NUIs when compared with desktop computers 
that commonly offer a mouse and a keyboard to interact with the interface (Steinberg, 2012). This is 
because they allow the interaction with content and applications using touch, stylus, gestures, pressure, 
and other ways to provide input. Functionalities of interactive surfaces also commonly include the 
provision of a work space that offers multiple direct inputs, so users can manipulate digital content with 
fingers or through physical, trackable objects (e.g., pens, gloves, fiducial markers) while they 
communicate via speech, facial expression, and gesture (Ardito et al., 2015). Less explored 
functionalities of these devices include the unique opportunity to automatically capture students' digital 
footprints that can be analyzed to make normally ephemeral f2f interactions persistent and (once 
rendered in appropriate form) "visible." Their intrinsic multi-user capabilities can assist in enhancing 
collocated exploration, discussion, and sense-making of LA indicators. Furthermore, Oviatt's (2013) 
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research argues for the critical role of creative sketching in learning, placing renewed emphasis on 
digital pens and surfaces. 


These underexplored opportunities motivate the need to define key dimensions of a new design space, 
where surface technology and LA tools can meet to address f2f learning challenges. We propose that the 
articulation of such an explicit design space should be helpful for both researchers and designers. Firstly, 
a design space serves as an instrument for making sense of past research, coordinating future research 
by identifying potential uses of the technology and challenges, and avoiding unintended duplication of 
effort. Secondly, a design space should provide conceptual guidance for the development of new LA 
tools by clarifying the scope (e.g., target audience; desired functionality) or by targeting unmet needs 
and underexplored teaching and learning areas that the design space highlights. Designing and 
deploying LA tools using interactive surfaces and tangible devices requires a comprehensive 
understanding of interaction design and the possibilities that these technologies offer, not just for 
learning and teaching, but also for LA. A design space should also consider the pedagogical 
underpinnings, the different target actors, the teaching strategies, data sources, the potential impact on 
authentic scenarios, and the degree of maturity of research and development in this area. 


This paper presents a synthesis of conclusions drawn from an empirical analysis of current research and 
development, exploring ways to deploy LA innovations utilizing interactive surfaces and tangible 
interfaces in novel and creative ways. To describe the design space in this emerging area, we use a 
composite analytical framework, drawing principles from five sources: 1) a framework of classroom 
orchestration (Prieto, Dlab, Gutierrez, Abdulwahed, & Balid, 2011); 2) a framework to support the 
implementation of teaching practices in the classroom (Kaendler, Wiedmann, Rummel, & Spada, 2015); 
3) the actors commonly targeted by LA tools (Siemens, 2012); 4) the iterative process they commonly 
follow to use and respond to LA tools (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013); and 5) the 
impact of the LA deployment (Santos, 2016). We analyze the technological and educational aspects of 
our first-hand experiences as researchers/designers, and the small number of authentic deployments of 
LA utilizing different types of interactive surfaces that we have been able to identify. In parallel, our 
approach allows us to analyze the maturity of multi-user interactive surface technology and LA solutions 
by drawing a contrast between interesting pieces of research conducted in controlled lab conditions and 
authentic classroom deployments. The contribution of this paper is the discussion of our analysis results 
that helped us identify current trends, gaps, challenges, and pedagogical opportunities offered at the 
intersection of the applications of LA and the use of NUIs for supporting learning. 


The rest of the paper is structured as follows. The next section provides an overview about touch and 
tangible interaction, and a definition of orchestration technology and its links with LA. Section 3 presents 
the theoretical underpinning of our composite framework. Section 4 discusses case studies associated 
with the use of LA tools and techniques to NUI-based learning scenarios. This section also describes the 
cases using the framework, making an emphasis on particular orchestration challenges, pedagogical 
uses, and advantages of using interactive surfaces and tangible interfaces for LA purposes. Section 5 
discusses the application of our framework, the maturity, opportunities, and needs in this area. Section 
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6 concludes with a discussion of alternative uses of the framework and possible avenues of future work 
in the area of LA using surfaces and tangibles. 


2 BACKGROUND 


2.1 Touch and Tangible Surfaces in Education 

Surface computing is still a maturing technology, which has become a more natural alternative to 
traditional mouse and keyboard input by enabling multi-touch interaction using fingers, hands, or 
special pens (Brown, Wilson, Gossage, Hack, & Biddle, 2013). Similarly, tangible interfaces offer the 
additional advantage of allowing users to manipulate physical objects and thus receive haptic feedback, 
which is more naturalistic for tasks that require the experience of textures, modelling realistic spatial 
representations, or mastering visual-motor skills (Evans, Drechsel, Woods, & Cui, 2010; Schneider & 
Blikstein, 2015a; Schneider, Jermann, Zufferey, & Dillenbourg, 2011). This shift in input technology has 
opened the interaction space, allowing a wide range of new collaborative and ubiquitous applications, 
especially for tasks more effectively performed face-to-face (Olson et al., 2002). 

Multi-touch and tangible interfaces often allow learners to interact directly with objects instead of using 
indirect input devices such as the keyboard, the mouse, or pointers. Performing physical actions on 
physical or digital objects allows students to be more easily aware of what others are doing. 
Additionally, students can combine the advantages of the physical setting provided by traditional f2f 
meetings with the possibilities that a digital environment can offer (Higgins, Mercier, Burd, & Hatch, 
2011). Some example tasks that have been supported by multi-touch and tangible interfaces include 
group planning (Jermann, Zufferey, & Dillenbourg, 2008), diagramming (Frisch, Heydekorn, & Dachselt, 
2009), designing (Martinez-Maldonado, Goodyear, Kay, Thompson, & Carvalho, 2016a), data exploration 
(Abad, Anslow, & Maurer, 2014), brainstorming (Clayphan, Martinez-Maldonado, Tomitsch, Atkinson, & 
Kay, 2016), knowledge building (Baraldi, Del Bimbo, & Landucci, 2008), and information curation (Apted 
& Kay, 2008). However, whilst advancements in hardware have been rapid (e.g., through gaming 
products sensing movement and gesture as well as tablet technology), application software for large 
surface devices is still in its early stages compared with (for instance) the market of mobile devices. 

In terms of educational contexts, there has been great interest in using large interactive surfaces for 
supporting collaborative learning pedagogies (Higgins et al., 2011). Interactive white boards (IWBs) have 
been used to conduct whole class activities (Clayphan et al., 2016; Evans & Rick, 2014), both vertical and 
horizontal large touch screens have been used to conduct small group work (Kharrufa, Martinez- 
Maldonado, Kay, & Olivier, 2013), and multiple tablets have been interconnected to support tasks in 
pairs (Wang, Tchounikine, & Quignard, 2015), or to show a user interface just for the teacher (Kharrufa 
et al., 2013). Moreover, the use of tangible objects on surface interfaces has been regarded by 
practitioners and researches as a particularly important feature for the cognitive development of young 
students' coordination and 3D orientation (Dillenbourg & Evans, 2011). Tangible interfaces are 
promising for tasks that require the manipulation of objects, which is not possible in flat displays; 
examples have included narrative, biochemistry, and simulation systems (Dillenbourg & Evans, 2011). 
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There is increased interest in the digital affordances of surface devices to support handwriting and 
sketching (Oviatt, 2013). These are often more fluid ways for students to communicate and generate 
ideas, compared with the use of mice, and physical or on-screen keyboards. Another use of large surface 
devices has been collaborative data visualization, mostly for geospatial datasets (Abad et al., 2014), but 
there has been little attention to date on the processes of collaborative sense-making around 
educational data visualizations. 


In short, surface-based devices (and to a lesser degree tangible devices) are now mainstream products. 
However, compared to the massive investment in analytics from systems that mediate interaction 
between remote participants, there is little work exploiting surface and tangible analytics. Not 
surprisingly, the complexity of capturing, disambiguating, and interpreting f2f user actions makes this 
quite challenging. As with all analytics applications, the goal is not to try to record everything; rather, 
were it possible to capture and render persistent traces from normally ephemeral f2f interaction, what 
would be appropriate and desirable data to log? It is timely to consider the potential enhancements that 
LA could provide to f2f learning. 

2.2 Technology for Classroom Orchestration 

The metaphor of classroom orchestration was originally defined in terms of the real-time management 
of classroom resources, learning processes, and teaching actions (Dillenbourg & Jermann, 2010). This 
metaphor takes into account the variability and complexity of classrooms and the key role of teachers in 
adapting the available pedagogical and technological resources to help students achieve their intended 
learning goals (Dillenbourg et al., 2011; Roschelle, Dimitriadis, & Hoppe, 2013; Twiner, Coffin, Littleton, 
& Whitelock, 2010). Orchestration in the classroom can be defined as a loop of awareness and 
regulation: the teacher monitors the state of the classroom, compares it to some desirable scenario, and 
performs actions to reach a more productive state (Dillenbourg et al., 2011). This loop is very similar to 
the one described by Soller, Martinez, Jermann, & Muehlenbrock (2005) applied to fully computer- 
mediated learning systems and the analysis of student data. In both cases (for computer-mediated and 
f2f learning) there is scope for producing indicators of learning processes to analyze and provide 
support, or enhance awareness of the learning processes. 

Importantly, the metaphor has been further embraced by other researchers to explain several other 
activities that need to be attended to before and/or after the actual deployment of learning tasks, not 
only in the classroom, but also in online or blended learning scenarios (Prieto et al., 2011). The 
community has also defined the mechanisms and technologies to support the orchestration activities 
(Tchounikine, 2013). Orchestration technology may support the management of the orchestration or 
some part of it. This includes, for example, systems that help teachers manage the class workflow 
(Martinez-Maldonado, Clayphan, & Kay, 2015a), enhance their awareness (Gutierrez Rojas, Crespo 
Garcia, & Delgado Kloos, 2012), track student progress (Rodriguez-Triana, Martinez-Mones, Asensio- 
Perez, & Dimitriadis, 2014), or provide informed feedback (Gutierrez Rojas, Crespo Garcia, & Delgado 
Kloos, 2011). Other orchestration approaches have been focused on supporting teachers to deploy their 
learning designs (Dimitriadis, Prieto, & Asensio-Perez, 2013), guiding reflection sessions (Do-Lenh, 2012), 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


28 


JOURNAL OF LEARNING ANALYTICS 


S8LAR 


(2017). Learning analytics for natural user interfaces. Journal of Learning Analytics, 4(1), 24-57. http://dx.doi.Org/10.18608/jla.2017.41.4 

re-designing after the activity is completed (Martinez-Maldonado, Kay, Yacef, Edbauer, & Dimitriadis, 
2012; Roschelle et al., 2013), and providing tools that can be flexibly re-configured by the teachers 
themselves to serve specific orchestration purposes (Tchounikine, 2013). 


The metaphor of classroom orchestration also promotes the use of data to support state awareness and 
workflow manipulation. According to Dillenbourg et al. (2011), the technology itself does not need to 
perform complex analysis or automated actions; instead, it should provide just the key information 
about the classroom state, leaving the diagnosis of such data to the teacher. This coincides with some of 
the objectives of LA innovations aimed at bringing human judgement into the analysis loop (Siemens & 
Baker, 2012). Moreover, the metaphor has also been extended by the notion of distributed 
orchestration (Sharpies, 2013), considering that other actors of the learning process besides the teacher 
can also be responsible for part of or all the orchestration tasks. Thus, orchestration may be applicable 
to self-managed learning scenarios where students are responsible for orchestrating key aspects of their 
own learning experience (Roschelle et al., 2013). 


The orchestration metaphor empowers teachers as drivers of classroom activities and advocates for the 
use of simple technologies that may have important effects. The effectiveness of orchestration and the 
extent to which teachers can respond to the ways students perform their tasks is critical because it 
directly impacts student activity, and therefore, their learning. Our work takes an approach based on 
orchestration because it is a dynamic perspective that considers authentic issues arising in the 
classroom, usually affected by unanticipated processes and contingencies. In contrast with learning 
theories that focus on cognitive aspects, orchestration is concerned with practical issues and tasks not 
directly linked with learning but that can shape learning. This makes orchestration very relevant for 
deploying LA tools in authentic learning settings: LA could play a key role in supporting f2f and blended 
learning activities. To achieve this, a clear understanding of orchestration activities is needed to create 
effective LA solutions in those f2f settings where teachers or students need to adapt quickly to 
unexpected problems. 

3 THE COMPOSITE FRAMEWORK 


The composite framework we used to analyze the current research and deployments that combine LA 
tools and interactive surfaces is defined by five dimensions: a) a set of orchestration activities to which 
the LA tools provide support, b) the phases of the pedagogical practice that are supported, c) the target 
actors of the LA, d) the level of impact of the LA innovation, and e) the extent of iteration of the LA and 
pedagogical processes (see Figure 1). The composite framework is a 5-dimensional matrix that can 
categorize a wide variety of LA deployments. Each of these dimensions, and their theoretical 
underpinnings, are described in the rest of this section. 

3.1 Orchestration Activities 

Prieto, Dimitriadis, Asensio-Perez, and Looi (2015) developed a framework that identifies four 
orchestration activities. For the first dimension of our analysis framework, we considered these four 
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orchestration activities that either teachers or students perform (see Figure 1, a). LA solutions can be 
created to support the actors in performing such activities. The four orchestration activities are design, 
management, adaptation, and awareness. 



Institution / Degree / Subject / Year cycle / Class / Small group / Individual 


IMPACT levels (d) 


Extent of the 
ITERATIVE PROCESS (e) 


Within a session 


kni) 


Across sessions 


Figure 1: A combination of frameworks creates this 5-dimensional framework, which we used to 
analyse the current state of learning analytics applied in contexts where interactive surfaces and 


3.1.1 Design 

Learning design includes the preparation of educational materials, pedagogical approaches, social 
dynamics, tasks, scripts, strategies, and any other resources needed to create learning opportunities for 
students (Goodyear & Retalis, 2010). Teachers commonly have a crucial role in learning design and co¬ 
design (Ertmer, Parisio, & Wardak, 2013; Tracey, Hutchinson, & Grzebyk, 2014). There may also be other 
actors specialized in learning design, particularly in higher education (Tracey et al., 2014). Alternatively, 
students can also design or co-design their own learning tasks (Goodyear & Ellis, 2007). The design 
process is not necessarily linear, as design and planning can co-occur while the actual activity unfolds or 
after it is completed (Prieto et al., 2011). In terms of LA, awareness and/or analytical tools may support 
fine-tuning of learning designs by providing visualizations of student data, indicators about how planned 
tasks actually occurred, or insights from the community of practice. In short, although learning design is 
an activity that commonly occurs before the actual learning activity, it can also co-occur while the 
student's activity unfolds (Goodyear & Dimitriadis, 2013). As a result, LA tools can provide support to 
teachers before, during, or after the learning activity for them (or instructional/learning designers) to 
revise the enactment of the design in runtime or make more permanent, substantial changes to the 
learning design that can have an impact on future learning sessions. 

3.1.2 Management 

Management refers to the coordination of the ongoing teaching process and/or the self-regulation 
during the enactment of the learning activity (a learning session). This includes the management of time 
for each student's task (e.g., the class duration), task distribution, and social arrangements. In short, this 
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activity is focused on the coordination of the workflow of the learning activity. For the case of f2f 
learning activities, the management is commonly performed at a classroom level. But this could also be 
virtual classroom management for online/blended activities. This regulation can be performed through 
social interaction (e.g., the teacher directing the flow of the class or students managing their own 
workflow based on feedback from LA systems) or be partly handed over to some computer controlled 
mechanism (Prieto et al., 2011). LA tools can support the actors responsible for the management of the 
learning processes by providing, for example, key information about the execution of the workflow so it 
can be modulated or reconfigured according to the demands of the activity. A LA tool could also be 
designed to support reflection for teachers so they can understand what went well and what went 
wrong during the enactment of the learning activity in order to modify its management for future 
sessions. 


3.1.3 Adaptation 

Adaptations or interventions are often needed for the designed learning activities and the original plan 
in order to respond to unforeseen or extraneous situations (for example, technology breaking down, 
students not arriving on time). Adaptations to the original plan may also be needed to take advantage of 
emergent learning opportunities (for example, when the teacher modifies the order of the tasks or skips 
some tasks because of the status or overall progress of the class). When the learning activity depends on 
the learning technology, the adaptation can be strongly influenced by the capacity of that technology, 
the class script, or the learning activities to be flexibly adapted to unexpected events and the emergence 
of new tasks (for example, some tools do not allow making changes to the script or design on the fly). 

Adaptation can also include the actors creating improvised tasks or adapting the planned tasks during 
the enactment, even in cases where the technology does not provide an elegant solution. Similarly, the 
systems can offer flexible functions to handle those adaptations. LA tools can support this process by 
providing teachers or students with key information that would allow them to manually intervene or 
adapt specific learning tasks, or for the learning system to automatically adapt the tasks to particular 
student's needs or provide automated interventions to tune the order or the approach to the tasks. 
Other analytics approaches such as predictive modelling or clustering could be applied to provide hints 
to the students or teacher before an unforeseen event occurs. For example, LA tools could automatically 
warn the teacher in case some unexpected event in the classroom is detected. As a response, the 
system may be able to propose optimal ways of group formation depending on the students in the class, 
automatically detect disengagement so the teacher can vary the tasks, or detect when a group has 
finished a task so they can advance to the next tasks. 

3.1.4 Awareness 

Awareness, along with formative/summative assessment tools, is clearly critical for orchestrating 
learning and relevant for visual LA innovations. Awareness includes the process aimed at getting insights 
about what is happening in the learning situation. While "awareness" of different sorts pervades all 
sense-making activity around data (Verbert et al., 2013), for this activity we focus on those awareness 
mechanisms particularly linked to the students' learning activity. Examples of learning activity data 
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include information about learning progress, student's logged actions, and formative and summative 
assessment results. 


Awareness support tools can, for example, enhance student awareness about their own work (self- 
awareness), about their collaborative work (group awareness), about the outcomes of others (peer 
assessment), or provide key information to the facilitators (teacher's awareness). LA tools may provide 
key insights into students' learning processes so actors can modify their teaching strategies, the 
provision of feedback, the pedagogical approach, or the students' learning strategies. These can be 
simple tools such as basic visualizations of group progress, more complex student modelling, or 
predictive approaches. 

3.2 Temporality: Phases of Pedagogical Practice 

The second dimension is derived from the Implementing Collaborative Learning in the Classroom (ICLC) 
framework by Kaendler et al. (2015) pointing to teacher competencies needed across the 
implementation phases of learning strategies in classroom sessions. It defines five teacher 
competencies: planning, monitoring, supporting, consolidating, and reflecting, which span three phases 
of teaching practice: pre-active, inter-active, and post-active (Figure 1, b). The authors map planning to 
the pre-active-phase; monitoring, supporting, and consolidating to the inter-active phase; and reflecting 
to the post-active-phase. Although teacher competencies could be matched with the orchestration 
activities described above, the metaphor of orchestration is not only concerned with the ability of the 
teacher to perform tasks according to professional knowledge. Rather, it is concerned with how 
different types of technology can support teachers, or students themselves, to manage multi-layered 
activities in a multi-constraint context (Dillenbourg et al., 2011). Additionally and very importantly is the 
idea that almost all orchestration activities can be relevant before, during, or after the activity (Prieto et 
al., 2011). For example, planning and learning design commonly occur in the pre-active phase, but it can 
be that a teacher has to adapt the intended design on the fly, or accomplish some re-designing work in 
the post-active phase. 

In summary, by combining both frameworks, we can map surface and tangible-based LA in terms of 
what orchestration support they provide and when. We have not yet specified for whom, which is 
considered next. 

3.3 Target Actors 

LA solutions can be oriented towards different actors of the learning process, including students, 
teachers, intelligent agents, administrators, etc. (Chatti, Dyckhoff, Schroeder, & Thus, 2012). At the 
same time, LA studies can be conducted for research, or prototype system-design purposes, without 
being deployed in authentic learning scenarios. LA can also support learning designers to make informed 
decisions based on evidence about changes that the course may require. Therefore, to understand the 
design space of LA in a specific area, and its degree of maturity in terms of real deployments, we should 
differentiate the actors being targeted as end-users of the LA tools. 
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Based on the users targeted by the current deployments covered in our analysis (e.g., who will make use 
of the LA outputs), we divide the target users into three groups (Figure 1, c): students, educators 
(including lecturers, tutors, learning designers), and researchers (including individuals and the 
community of research). 


3.4 Impact Levels of Learning Analytics 

The previous dimension referred to the actors who are the actual users of the LA tools. But LA can 
produce an indirect benefit in other aspects of the educational process (Verbert et al., 2013). For 
example, a dashboard can be targeted to teachers but the impact of using the dashboard can be 
translated into an improvement of student learning, or prompt the modification of the learning design 
of the following course (making a wider impact on the course or other students and teachers). As a 
result, the impact dimension refers to the different levels on which the LA tool can have an impact. 
Buckingham Shum (2012) defined three layers of LA impact. The first layer covers most of the case 
studies discussed in this paper, focused mostly on classrooms and small learning communities, where 
teachers and students are the main stakeholders. Buckingham Shum (2012) referred to this layer of LA 
research as the micro-layer. The meso-layer is focused on the institution (e.g., making an impact at a 
program level or adopting some LA framework by a whole institution), whereas the macro-layer looks at 
the broader society (for example, at a level of the entire educational system within a country). 


Nonetheless, it is not only challenging to define the boundaries between layers but also to isolate the 
impact of a tool on specific stakeholders. Given that the purpose of our framework is to understand the 
design space and the maturity of a specific area in LA, we are interested in narrowing down the intended 
levels that the LA tool may have an impact on to specific levels of stakeholders. Based on the layers 
identified by Buckingham Shum (2012) and the current deployments covered in our analysis, we divide 
the levels of impact of LA into six groups (Figure 1, e): individual, small group, class, which are associated 
with the micro-layer; subject (one edition of one subject), year cycle (including multiple editions of the 
same subject), which can be in between the Micro and Meso layers; course and institution, which are 
associated with the meso-layer. 

3.5 Extent of the Iterative Process 

The fifth and last dimension introduces the notion of iteration at two extents: within a 
(learning/analytics) session and across sessions (Figure 1, e). We will use the word session to refer to the 
continued learning activity that students perform either face-to-face or online. Examples of f2f sessions 
can be a classroom tutorial, a lecture, a small-group meeting, or an experimental trial. As a result, 
iteration within one session refers to the iterative process of LA support within each pedagogical phase. 
This has been described by Verbert et al. (2013) as the process users follow to have access to data 
(1. awareness); ask questions and assess the relevance of the data (2. reflection); answer questions, 
getting new insights (3. sense-making); and induce new meaning or behavioural change (4. impact). This 
four-stage iterative process occurs while users interact with a LA tool in a given phase. Iteration across 
sessions is concerned with the workflow as the phases ( pre-active, inter-active, and post-active) are 
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repeated over multiple sessions. This is crucial so that LA tools can provide support activities spanning 
multiple sessions. 


In summary, the composite framework considers that the pre-active, inter-active, and post-active 
phases form a linear workflow for one specific session (e.g., a classroom session, an experimental trial, 
an online task). Each orchestration aspect can be supported in any of these phases (e.g., planning is not 
restricted to the pre-active phase, but can occur in the inter-active and post-active phases). LA support 
can be targeted at different actors in each phase, and across phases or sessions. Finally, the LA tools can 
have a particular impact on other stakeholders at different levels, from the individual student to the 
whole institution. 


4 ANALYSIS OF CASE STUDIES 


In this section, we analyze a series of case studies of LA applications that use interactive surfaces or 
tangible devices to support different orchestration activities. Table 1 presents an overview of the design 
space defined by the dimensions of our framework. We have analyzed our own current R&D exploring 
LA associated with the use of interactive surfaces and tangible devices, and a small number of 
deployments by other researchers that we identified as relevant. As far as we are aware, our list of cases 
is the first joint effort to bring together LA innovations and NUI-based learning settings. The table maps 
the projects analyzed (column 1), the orchestration activities addressed (2-5), the pedagogical phases 
supported (6-8), and whether they involve certain levels of iteration (9-10). The actors targeted in each 
deployment are represented by letters: E for educators, teachers, tutors, and learning designers; S for 
students; and R for researchers. Beyond the dimensions of the composite framework, in the case studies 
we seek to identify the forms in which the data is communicated to the actors, such as whether it is 
presented in a raw format (e.g., statistics, algorithms results, patterns), through visual representations 
(e.g., dashboards, visualizations, alerts, notifications), or by direct automated actions. We also pay 
attention to the topology of LA tools classified by the type of information they offer, including 
information about 1) the task/class progress, 2) student interaction, 3) quality of the students' solution, 
and 4) learning (including conceptual change, learning to collaborate, or learning about the process). 

In the following subsections, we provide a concise description of our first-hand experiences from seven 
case studies (the first seven rows in Table 1). These cases serve to illustrate how the dimensions of the 
composite framework are interwoven to help understand the technologies used and pedagogical 
aspects tackled by the LA solutions. To facilitate the presentation of the cases, these are grouped by the 
main actors targeted in each (teachers/learning designers, students, and researchers, respectively). 
Lastly, we briefly describe other LA applications where some sort of surface technology has been used 
(last four rows of Table 1). 
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Table 1: Analysis of the Current Design Space of Learning Analytics Applications Utilizing Interactive 
Surfaces. Target Actors: E=Educators, S=Students, R=Researchers 


Project 

Design 

Orchestration Activities 

Awareness Adaptation Management 

Pedagogical 

Phases 

Pre Inter Post 

Impact 

Iteration 

Within Across 

1- MTFeedback (Martinez- 
Maldonado et al., 2015b) 


E 



V 


Group & 
Class 



2- Analytics for redesign 
(Martinez-Maldonado et al., 2012) 

E 

E 


E 



Year cycle 



3- CoCo design table (Martinez- 
Maldonado et al., 2015c) 

E/R 




V 


Subject 

V 


4-Navi surface 

(Ckarleer et al., 2013) 


E/S 





Individual & 
Subject 

S 

S 

5- LARAe.TT 

(Charleer et al., 2015) 


E/S 


E/S 

S 

V 

Individual & 
Subject 

S 

S 

6- Co-located eye-tracking 
(Schneider et al., 2015) 


R 



s 


Individual & 
Group 



7- Motion sensors 

(Schneider & Blikstein, 2015) 


R 



V 


Individual 



8- Script awareness tool 
(Martinez Maldonado et al., 

2015a) 



E 

E 

s 


Class 



9- Tinker lamps 

(Do-Lenh, 2012) 


E/S 





Group & 
Class 



10- Monitoring tablets 

(Wang et al., 2015) 


E 





Class 



11- Learning catalytics 

(Schell et al., 2013) 


E 



s 


Subject 




4.1 Supporting Awareness for Teachers 

We start by describing two case studies of analytics support for enhancing teacher awareness. 

4.1.1 MTFeedback: Helping teachers track and prioritize groups with targeted feedback 

The first case study consisted of providing support to enhance teachers' classroom awareness and 
assessment on the fly (inter-active phase). The pedagogical intentions of the teachers were that 
students could engage in collaborative discussions and visually represent their proposed solutions to 
challenging problems. The teacher conducted this activity face-to-face to support students and provide 
direct feedback to promote verbal discussion and argumentation. The setting used was the 
MTCIassroom (Figure 2, left). This is a multi-surface classroom environment composed of 4-5 large 
interconnected tabletops and three vertical displays. Each tabletop was enriched with a Kinect sensor 
that differentiates individual touches. This allows for the capture of an identified log of student actions 
at each table. Six teachers and more than 300 students were involved in a series of realistic studies 
conducted during three regular semester courses. Three types of tasks were facilitated by the tabletops: 
collaborative concept mapping, brainstorming, and scripted group meetings. All the tabletops and 
vertical displays were controlled by a teacher's tablet-based dashboard (Figure 2, right). This also 
showed visualizations that conveyed student information in two dimensions: individual participation and 
group progress. It also showed notifications from the MTFeedback subsystem. This analyzed student 
artefacts in the backend to generate both positive and negative notifications according to the groups' 
misconceptions or underperformance, automatically identified based on thresholds set by the teacher. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


35 



















JOURNAL OF LEARNING ANALYTICS 


S8LAR 


(2017). Learning analytics for natural user interfaces. Journal of Learning Analytics, 4(1), 24-57. http://dx.doi.Org/10.18608/jla.2017.41.4 



Orchestration 

controls 


Figure 2: Left: An ongoing small-group session in the MTCIassroom. The teacher is holding a tablet- 
dashboard while providing feedback to one team. Right: The dashboard showing visualizations of 

participation for four groups. 


Small group 
visualisations 


« Individual tables 
controls 


Empirical evaluations studied if the visualizations and notifications shown in the dashboard effectively 
supported teachers' iterative LA process within a session by enhancing their classroom awareness and 
thus allowing them to take more informed decisions when selecting the groups that required more 
attention (Martinez-Maldonado, Clayphan, Yacef, & Kay, 2015b). Results indicated that the system 
helped to capture traces of student activity seamlessly, thus allowing the generation of live 
visualizations and notifications for the teacher. The deployment of the teacher's dashboard on a tablet 
allowed free mobility to the teacher while having access to control and monitoring tools. The 
visualizations and notifications allowed teachers to attend to groups that needed immediate support 
and provide formative and/or corrective feedback, which translated into students' conceptual changes. 
This also made an impact on teacher awareness in terms of gaining insights about the status of the 
whole class. Table 1, row 1, presents an overview of how this case study can be described based on the 
different elements of the framework. Figure 3 and Table 1 (row 1) show how each element of the 
framework categorizes this case study. 


MTFeedback 


Awareness 


Educators 



Inter¬ 

active 




Class / Small group 


Figure 3: Representation for the case "MTFeedback" according to the elements of the framework. The 
teacher used the system to gain awareness about small group interaction in the classroom. 


ISSN 1929-7750 (online). The Journal of Learning Analytics works under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) 


36 











JOURNAL OF LEARNING ANALYTICS 


S • LAR 

SOCIETY for LEARNING 
ANALYTICS RESEARCH 

(2017). Learning analytics for natural user interfaces. Journal of Learning Analytics, 4(1), 24-57. http://dx.doi.Org/10.18608/jla.2017.41.4 

4.1.2 Learning Analytics for Redesign 

For the second case, the situation was also associated with the MTCIassroom for conducting tutorials; 
however, this study focused only on one teacher designing and then re-designing 1-hour tutorials for 
two different subjects (business and management) in two consecutive university semesters. The study 
provided LA support in two forms, iterating across two sessions (Martinez-Maldonado et al., 2012). First, 
the teacher had access to a set of visual analytics in the post-active phase of a classroom session for 
semester 1 of an undergraduate subject. The aim of the visualizations was to enhance teacher 
awareness and helping her assess how the initial intentions played in the classroom. Second, in the pre¬ 
active phase of the next class session (semester 2), the visualizations provided insights into the aspects 
of the learning tasks that needed to be redesigned (see representation of the case according to the 
framework in Figure 4 and refer to Table 1, row 2). The first tutorial involved 236 students distributed in 
14 classroom sessions. The second involved 140 students distributed in 8 sessions. The goals and the 
topic of both tutorials were similar: to promote discussion and deep understanding of political dynamics 
for students to learn how to address organizational issues. Both tutorials had a similar macroscript, 
which consisted of two small-group concept-mapping tasks. The captured data included application logs, 
snapshots of the evolution of each group's concept map, and teacher actions to advance the class 
according to her script. 


Semester i 


Design 

Awareness 




Post¬ 

active 


Semester 2 


Design 




Pre¬ 

active 




Teachers 


Year cycle 


Figure 4: Representation of the Learning Analytics for Redesign case according to the elements of 

the framework. 


Four semi-structured interviews were held with the teacher after the tutorials to capture their 
intentions and reflections. The first interview served to elicit the teachers 7 intentions that the 
information captured by the MTCIassroom could inform about. These intentions were grouped into 
three categories: the class script progress (A), student participation (B), and student achievement (C) in 
all sessions. In the second interview, the LA support was presented to the teacher in the form of 
visualizations (graphs), workflow diagrams, and raw numerical results about each of the three 
pedagogical intention categories. This supported teacher reflection in the post-active phase of the first 
macro-level iteration of the LA cycle (Martinez-Maldonado et al., 2012). The next two interviews focused 
on capturing the teacher's re-design decisions as part of the pre-active phase of the next iteration. 
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Regarding the class script (A), for example, the teacher was provided with a fuzzy workflow diagram (see 
Figure 5). She identified that in most tutorials, students spent too much time on the first task, not 
leaving enough time to complete the second task. Concerning student participation (B), a bar chart was 
shown to the teacher, indicating that within most groups participation had not always been equally 
distributed. A third example (for category C) is illustrated by the results from a correlation analysis, 
which suggested that a hierarchical concentric arrangement of student concept maps was associated 
with achieving better solutions. These insights were informative for the teacher in re-designing the 
tutorials. For the next tutorial sessions, the teacher provided an initial scaffolding solution for students 
to progress more quickly and focus on the subsequent higher-level tasks. The teacher also developed a 
strategy to encourage all students to use the tabletop, and to follow a specific concentric layout. 
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Figure 5. Planned time limits for 5 tasks (top row) and the enactment of the design for 14 tutorials 

(bottom row) (Martinez-Maldonado et al., 2012). 

In this study, the surface devices allowed the automated collection of classroom evidence. The data was 
exploited to generate visual and non-visual information to help the teacher compare her planned 
intended goals with how they actually played in the classroom. This example illustrates the synergy 
between surface technology and LA to provide continued iterative support to teacher awareness and 
planning across sessions, which can make an impact on her whole year-long teaching cycle. 

4.2 Analytics for Learning Designers 

A recent report documents problems caused by a lack of mutual awareness among users of computer- 
mediated learning design systems (Nicolaescu, Derntl, & Klamma, 2013). One of the functionalities of 
using large surfaces is that they invite all team members to interact with the shared device, making their 
actions visible. The next subsection describes a case study of analytics support for learning design. 

4.2.1 CoCoDes: Collaborative educational design 

This case study consisted of supporting design in the pre-active phase using design analytics that can 
make an impact on learning designers while they design and plan the tasks for a whole subject. The 
specific goal of this study was to understand how surface technology and minimalist visual analytics can 
support high level learning design. Figure 6 and Table 1 (row 3) present how each element of the 
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framework categorizes this case study. The setting was the Design Studio (Martinez-Maldonado et al., 
2015c). Figure 7 shows this multi-surface space providing a set of digital and non-digital tools, including 
a tabletop, an IWB, tablets, a white-wall, a dashboard, and various paper-based materials. The tabletop 
and the IWB run an application called CoCoDes. It offers a large interface customized to support rapid 
construction of candidate designs as part of the conceptual design stage of university courses. The tool 
shows a flipped timeline where users can arrange learning tasks on a weekly basis. This allows the 
manipulation of iconic digital objects to configure spatiotemporal characteristics of learning tasks and 
their workflow. 


CoCo Design table 


Design 


Adaptation 

Management 

Pre¬ 

active 




Educators 


Year cycle 


Figure 6: Representation of the CoCo Design Table case according to the elements of the framework. 
One iteration shows a number of teachers and learning designers redesign candidate subject designs 
and use an analytics dashboard to compare them back to back. 


The dashboard shows live visualizations of the candidate designs created in the surface devices. This 
information includes a list of the learning tasks added to each candidate design, a pie chart that shows 
how students' time would be divided among learning spaces (face-to-face and online), and a histogram 
showing the student's weekly workload (see Figure 7, right). The goal of presenting a dashboard with 
visualizations of multiple candidate designs is to support teachers' high-level comparison and promote 
understanding of the impact of substituting certain learning tasks for equivalent tasks on students' 
workload and direct contact time. 


Four teams of three teachers and learning designers participated in an observational lab study. The goal 
of each team was to produce two high-level candidate designs of a university course, satisficing some 
competing design goals. Results of the study showed that the dashboard was one of the features that 
was most valued by participants. It provided an overall view of the tasks within each design and helped 
most groups in keeping themselves on track toward their design goals by having continuous access to 
indicators of their designs. Moreover, participants valued the combination of large devices to have a 
view of the designs, smaller sized tablet devices to seek information as needed, and the dashboard to 
keep aware of the changes to their designs. 
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Figure 7. Left: A group of designers looking at the dashboard while designing two candidate designs (A 
and B) in the Design Studio. Right: The dashboard showing a) the tasks included in each design (left), b) 
the proportion of tasks by learning space (middle), and c) the weekly distribution of student time 

between online and f2f work (right). 


4.3 Collaborative LA Data Exploration 


Collaborative tools have been used to help small groups keep a shared view and articulate their insights 
more fluidly than with single-user displays. Surface devices can be used to support collaborative 
reflection on educational data. Next, we describe two case studies of collaborative LA exploration. 


4.3.1 Navi Surface 

This case study aimed to support students by enhancing their awareness about their achievements to 
help them self -regulate their own learning. The approach relies on a student dashboard that can be used 
in the inter-active and/or the post-active phases (see Figure 8). Figure 9 (left) and Table 1 (row 4) 
present how each element of the framework categorizes this case study. The third author and his 
colleagues used the notion of badges to create Navi Surface (Charleer, Klerkx, Odriozola, Luis, & Duval, 
2013). Badges are used to abstract important aspects of student learning processes, including intended 
learning outcomes and produced artefacts such as blog posts and shared documents. 



Figure 8: Students using Navi Surface in pairs to explore their achievements through a 

collaborative badge visualization. 
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Navi Surface is a tabletop-based tool that allows teachers and students to navigate student 
achievements for a university course. Users can navigate through the tool to get more information 
about how and why badges were awarded to which students, based on the learning traces captured 
during the course. Multiple items can be accessed simultaneously, enabling group interaction with the 
data. The teacher can guide the process, for example, by dragging relevant course goals onto the 
interface to promote discussion about what students have achieved, while students can also interact 
and steer the conversation. 


Navi Surface was evaluated with 14 students (4 groups of 2, 3, and 4 members) who used the tool in 
groups and individually, and were able to access their badge data and that of others. Preliminary 
observations showed that the interface promoted engagement, group interaction, and evaluation of 
achievements. This can be explained as follows: Most dashboards provide a single-user experience, 
requiring motivation (either intrinsic or extrinsic) from a student to access the LA data. Interactive 
tabletops can create a more inviting environment and facilitate a shared experience for students and 
their teachers. The tabletop played a key role as a catalyst for discussion, and participants reported the 
approach as a fun way to interact collaboratively with LA data. By contrast, when students used the 
tabletop alone, a more hesitant interaction with the LA data was observed. These observations suggest 
that the collaborative nature of the tabletop device promoted social discourse for the exploration of 
student data. Future work is required to test if this increases the chances of constructive reflection by 
students about their achievements. The setup supports awareness and reflection about personal 
achievement during the inter-active (during course sessions) and post-active (for evaluation purposes) 
phases. Sessions can be repeated during the course to increase awareness of student progress and 
achievements across sessions. 


Navi Surface 
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Figure 9. Representation of the Navi Surface and LARAe.TT cases according to the elements of the 
framework. In both cases, the systems enhanced reflective discussion among students and teachers 

using large displays for several sessions. 

4.3.2 LARAe.TT 


The second case study in this section includes the use of LARAe.TT (Charleer, Klerkx, & Duval, 2015). 
Similar to Navi Surface, this tabletop tool aims to support student awareness and reflection in the inter¬ 
active and post-active phase, particularly for inquiry-based learning (IBL) activities. In IBL, teachers 
encourage learners to pose questions and formulate hypotheses about a given topic, accomplishing 
independent investigations to support their conclusions. LARAe.TT visualizes the paths that students 
follow through their inquiry-based learning activities. The tool is grounded on an IBL process model. 
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which distinguishes six phases: problem identification, operationalization, data collection, data analysis, 
interpretation, and communication. Thus, students assume an active role in regulating their own 
learning as they follow their individual paths. LARAe.TT allows students and teachers to discuss and 
retrace individual steps taken by students. They can look up related content such as hypotheses 
formulated, evidence gathered, and so forth. 

Figure 10 shows the LARAe.TT interface, with the visual representations of student learning paths in the 
centre. The application provides a series of drop zones that allow students and teachers to drag 
activities to see more detail in the form of text or pictures that evidence student activity for a particular 
IBL phase. Dragging a student name into a personal drop zone (the coloured squares in Figure 10) allows 
students to explore and filter their data according to the positions of participants at the tabletop. 



Figure 10. LARAe.TT activities are shown in the centre of the screen. The top drop zone lets users 
expand an activity to get more detail. Each user has a coloured, personal drop zone for highlighting 

activities. 


LARAe.TT was presented to and evaluated by 15 participants (teachers, students, and researchers) at a 
workshop. The evaluation explored how the tabletop application can assist both students and teachers 
during the IBL process. It showed potential to facilitate students assessing their own progress and 
managing the distribution of their work. LARAe.TT can not only help students explore personal 
achievements, but can also let them compare, reflect on, and learn from the activities of their peers. 
Teachers, on the other hand, can invite students to the tabletop to initiate a discussion, intervene, 
discuss progress, ask for clarification and reasoning, assess activities, and point out peer activities for 
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comparison and inspiration. As such, teachers and students can use the tabletop system during 
evaluation sessions to visualize the history of individual student activities across the entire course. Thus, 
LARAe.TT can provide support for teachers and students to gain better understanding of their individual 
learning paths at a subject level, from the initial sessions to the final evaluations. Figure 9 (right) and 
Table 1 (row 5) present how each element of the framework categorizes this case study. 


Overall, Navi Surface and LARAe.TT illustrate a very particular orchestration use for interactive surfaces 
to support reflection and post hoc assessment. The physicality of the tabletop and the design of the 
interface provide a unique opportunity to support collective f2f exploration of student data with the 
purpose of facilitating discussion between students and their teacher. 

4.4 Multi-Modal Learning Analytics for Researchers 

The previous case studies suggest that interactive surfaces provide opportunities to support students' 
f2f interactions and teachers' orchestrations. At the same time, they also provide researchers with a 
wealth of information to better understand the nature of social learning in the inter-active and post¬ 
active phases: researchers can use many data collection tools to capture student interactions as they 
learning new concepts by using cameras, microphones, motion sensors, mobile eye-trackers, galvanic 
skin response sensors, and emotion detection tools. We see interactive surfaces as environments where 
rich learning episodes can occur, which makes them ideal devices for using multi-modal sensors. We 
illustrate this idea with the two examples below. 


4.4.1 Mobile eye-trackers and joint visual attention 

This case study is about capturing a fundamental building block of student interaction: joint visual 
attention (JVA), known by developmental psychologists and learning scientists to be a pre-requisite for 
any kind of high-quality collaboration because it allows a group to build common ground to solve a 
problem effectively. The third author and his colleagues (Schneider et al., 2015) have developed 
innovative ways to capture JVA around interactive surfaces to provide measures about individual and 
group strategies and performance. Their methodology involves using fiducial markers (Figure 11) to 
remap student gaze onto a ground truth. Since the fiducial markers are part of the tangible interface, 
the interactive surface becomes an essential part of being able to collect and meaningfully analyze the 
eye-tracking data. Having both gazes on the same physical plane allowed the researchers to determine 
whether students were jointly looking at the same location at the same time. They found that the 
number of times that JVA is achieved is not only correlated with students' quality of collaboration, but 
also reflects higher performance on the problem-solving task as well as higher learning gains. This kind 
of data stream allows researchers to generate reliable footprints of collaboration quality, and separate 
productive from less productive groups of students. This data could potentially be collected in real-time 
to help teachers decide which groups need attention and which ones do not need help. 
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Figure 11: Two students analyzing a static version of a tangible interface. Red lines show the 
points used for remapping student gazes onto a ground truth (middle figure). 


For instance, those authors were able to identify different collaborative interactions by looking at the 
eye-tracking data in more depth (Schneider et al., 2016). For each moment of JVA, they identified which 
student was at this particular location first (i.e., who initiated it) and who was there with some lag (i.e., 
who responded to it). Thus, students in each dyad had a score (in percentage) describing how many 
moments of JVA they initiated and responded to. By taking the absolute difference of those scores, they 
obtained a measure (between 0 and 1) describing how groups distributed the responsibility of initiating 
moments of JVA. Groups with a score of 0.0 were perfectly balanced, while groups with a score closer to 
1.0 were imbalanced. This measure ended up being a significant predictor for the group's learning gains, 
and was found to represent (non-)productive collaborative dynamics in the transcripts. Imbalanced 
groups were more likely to have passive students who would generally agree with their partners, while 
balanced groups were more likely to have students who would challenge their partners and negotiate 
new knowledge in rich ways. 
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Co-located eye-tracking and motion sensors 


Educators 
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Inter¬ 

active 



Researchers 


Small group / Individual 


Figure 12. Representation of the Co-located Eye Tracking and Motion Sensors cases according to the 
elements of the framework. In both cases, the systems helped researchers gain an understanding of 
small group and individual performance while students collaborate face-to-face. 


This last example shows that one interesting aspect of multi-modal sensors is that they allow 
researchers to more easily go back and forth between qualitative and quantitative data, helping them 
more easily generate new measures at the sample level. The next step of this line of research is to look 
at videos augmented with gaze information (Figure 6 shows one frame of this kind of video) to support 
qualitative analysis of student interactions. This kind of analysis was previously difficult to conduct since 
it required researchers to position multiple cameras around a group to infer whether two students were 
simultaneously looking at the same location. Sensors can now provide this information to researchers, 
which can speed up the pace of qualitative work. Figure 12 and Table 1 (row 6) present how each 
element of the framework categorizes this case study. 


4.4.2 Motion sensors and students' physical mobility 

This last case study is about capturing another key aspect of f2f interactions: students' ability to use 
their physical body to express ideas and manage collaborative processes. These movements can be 
manually coded or captured using a motion sensor. For example, Schneider and Blikstein (2015b) used a 
Kinect sensor to collect data from a study conducted with 38 students interacting with a tangible 
interface, resulting in 1 million data points describing their body postures. They then fed this matrix into 
a simple clustering algorithm to obtain the following prototypical body positions (active, semi-active, 
and passive; see Figure 13). 

Not surprisingly, they found that the time spent by students in the "active" posture (left graph of Figure 
13) was positively associated with their learning gains while the "passive" posture (right graph) was 
negatively correlated with them. More interestingly, they found that the number of times students 
transitioned from one posture to another was the strongest predictor for learning. This suggests that the 
most successful students were the ones who not only acted, but also systematically stepped back to 
reflect on their actions and think about their next steps. With traditional qualitative approaches, it 
would have taken months to identify and code this kind of behaviour. Using sensors and unsupervised 
machine learning, it took a fraction of the time to isolate this productive learning behaviour. 
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Figure 13: The results of the clustering algorithm on students' body posture. The left centroid is 
active, with both hands on the table; the middle one is semi-active, with one hand on the table; 
the right one is passive, with both arms crossed. 


In the same study, researchers were also able to identify productive collaborative dynamics. After 
manually coding each student in the dyad as being the driver or the passenger of the group (i.e., the 
driver is the person who talks the most, decides what the group does next, and is generally more active; 
by default, the other person is the passenger), they found that drivers tend to use both hands equally 
(Figure 14, bottom graph) while passengers tend to use their dominant hand more (Figure 14, top left). 
Being able to identify roles was important in this study because the composition of the group was 
associated with different learning gains (Figure 14, top right). The authors created four groups by doing 
a median split on student grade point average (GPA). 

Not surprisingly, groups with two high-GPA students did well on the learning test (first boxplot) and 
groups with two low-GPA students did poorly (last boxplot). Interestingly, groups with a low-GPA driver 
did almost as well as groups with two high-GPA students (second boxplot). This is because more 
proficient students would be less pro-active in this situation, and would give the opportunity to low-GPA 
passengers to engage themselves intellectually. The high-GPA passenger would guide this process by 
gently suggesting ideas and by discriminating between fruitful and non-fruitful directions, which 
benefited both participants. Finally, groups with a high-GPA driver did almost as poorly as groups with 
low GPA-students (third boxplot): in this case, the high-GPA driver would do all the work in a very quick 
and efficient way, which would discourage the passenger and create a free-rider effect. This result 
shows ways to identify collaborative dynamics with sensors automatically, and suggests productive ways 
to engineer collaborative groups. 

In conclusion, results suggest that surface devices, augmented with multi-modal sensors, provide 
researchers with rich opportunities to collect massive datasets about student learning experiences. 
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Those datasets can then be mined using machine-learning algorithms, or used to augment videos and 
facilitate qualitative analyses of student interactions. Figure 12 and Table 1 (row 7) present how each 
element of the framework categorizes this case study. 




Driver 



Figure 14: Left side: averaged amount of movement generated by students' hands. Top right: Boxplots 
of the four kinds of dyads: driver/passenger with high/low GPA. The Y-axis shows the average learning 

gains of the dyads. 


4.5 Other Cases 

We also analyzed several other case studies. The first author and colleagues investigated the impact of 
showing the teacher visualizations about the enactment of the macro-script during a class session 
through a script awareness tool (Martinez-Maldonado et al., 2015a). This is the only example we are 
aware of that directly supported the orchestration activities of adaptation and flexibility to enhance the 
management of the workflow of a multi-surface classroom (see Table 1, row 8). 

Do-Lenh's (2012) work was very similar to the first case study described above. His system captured 
from each small group using multiple tangible tables in a classroom. Next, a public dashboard was 
displayed on an IWB for all students and their teacher to track their progress on the task, compared with 
the other groups (see Table 1, row 9). 
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Recent work by Wang et al. (2015) proposed similar visualizations of the progress of the task for 
students working with and sharing tablets (instead of tabletops) (see Table 1, row 10). Similar cases of 
LA applied to interactive surfaces are slowly emerging to support BYOD (bring your own device) 
strategies. An example is Learning Catalytics (Schell, Lukoff, & Mazur, 2013), which provides some visual 
analytics to teachers about student progress and their misconceptions while collaborating in the 
classroom using tablets or mobiles. The visualization tools offered by this system allow the teacher to 
keep their students on track throughout the duration of the class (see Table 1, row 10). 


5 DISCUSSION 


This synthesis of results identifies the degree of maturity, challenges, and pedagogical opportunities of 
LA and interactive surfaces. We discuss different aspects of the case studies presented above, the 
implications of defining this design space, the particular affordances of surface devices, and the kinds of 
analytics that look promising in supporting f2f collaborative learning challenges. 

A basic affordance of large surfaces is that (used well) they more readily support the ergonomic 
(perceptual, physical, cognitive, social) characteristics of groups than small surfaces. Therefore, it is not 
surprising that group work is a common denominator in most of the cases reviewed, but with the 
difference in some cases that they support novel kinds of interactivity, and critically, make them 
traceable. The case studies illustrate varied ways to capture student interactions, enabling teachers to 
provide enhanced feedback while orchestrating a classroom and permitting the collaborative 
exploration of student data. The combination of these technologies has the potential to open up new 
lines of research by allowing automatic processing and mining from large amounts of heterogeneous 
traces of f2f data (such as physical actions, gaze, body mobility, speech, etc.). Critically, these 
technologies are not only analytics tools for researchers, but show promise for providing real-time 
feedback on activity to students and educators. The people who constitute the learning system are 
provided with data about their own process whereas before they were the object of study by 
researchers, who were the only people with the tools to capture and render such data. Manually 
analyzing this kind of f2f data through more conventional video coding and observational approaches is 
time-consuming. As surface analytics matures, real-time analytics could become practical in authentic 
classroom settings at runtime. 

In the cases reviewed, interactive dashboards and visualizations were the most common ways to show 
educational data to educators and students. The focus was on providing information about the task 
(Charleer et al., 2015; Charleer et al., 2013; Do-Lenh, 2012; Martinez-Maldonado et al., 2015b; Wang et 
al., 2015) and activities of the class (Martinez-Maldonado et al., 2015a) progress (Case 1), students' 
interaction with the shared device (Case 2) (Martinez-Maldonado et al., 2015b; Martinez-Maldonado et 
al., 2012; Schneider & Blikstein, 2015b), the class design (Martinez-Maldonado et al., 2015c; Martinez- 
Maldonado et al., 2012), and, to a lesser extent, the quality of the students' solution (Case 3) (Martinez- 
Maldonado et al., 2012). Only two studies provided notifications (Martinez-Maldonado et al., 2015b; 
Schell et al., 2013) to the teacher during the inter-active phase to aid the decision making of the teacher 
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in the classroom. Finally, detailed and more complex analytics that give information about more abstract 
aspects of learning such as achievement (Charleer et al., 2013) and collaboration (Schneider et al., 2015) 
have mostly been lab studies (Case 4). 


The most suitable tasks for surface technology seem to be those that involve a combination of talk, 
discussion, manipulation of digital or physical objects in a spatiotemporal representation plane, and/or 
that require larger sized displays. The tasks in the case studies included collaborative concept mapping 
(Martinez-Maldonado et al., 2015b; Martinez-Maldonado et al., 2012), brainstorming (Martinez- 
Maldonado et al., 2015b), team meetings (Martinez-Maldonado et al., 2015b), data exploration 
(Charleer et al., 2015; Charleer et al., 2013), logistics training (Do-Lenh, 2012; Schneider et al., 2015), 
and a physiology challenge (Schneider & Blikstein, 2015b). The use of dashboards and visualizations in 
the classroom is still in its infancy. With the increasing use of digital surfaces in the classroom (e.g., 
tablets), it will be very common in the near future to see more implementations of systems that 
visualize key aspects of student activity and/or performance or simply visualize or notify them for cases 
where students are disengaged, underperforming, or not collaborating with their peers. This 
information could also be helpful for the students themselves to self-regulate their interaction and 
learning activities. The use of LA to support learning design is also an underexplored area of application. 


The data captured by interactive surfaces and the orchestration technology can also be valuable to 
facilitate teachers' reflections on their designs (Martinez-Maldonado et al., 2015c; Martinez-Maldonado 
et al., 2012), even if the time constraints of the class make it challenging to make big changes on the 
original plan, they can re-design for the following sessions. One case study (Martinez-Maldonado et al., 
2012) illustrated how orchestration support can be provided by LA at a macro level of iteration (across 
sessions), showing analytics about the planned curriculum compared to how it actually occurred. 
Regarding more complex, multi-modal analytics approaches, the challenge is to feed these data back to 
students (and teachers) to help them make better informed decisions and to support student 
collaboration. Gaze awareness tools where students in a remote collaboration can see the gaze of their 
partner in real time on the screen can be highly beneficial to students. This allows them to monitor the 
visual activity of their partner and anticipate their contributions, which leads to higher quality 
collaboration and higher learning gains (Schneider et al., 2015). Visualizations of individual learner traces 
on shared surface devices can help in bootstrapping dialogue between teacher and students. On the one 
hand, they allow learners to gain insight into the learning activities of themselves and their peers and 
the effects these have, while allowing teachers to stay aware of the subtle interactions in their course. In 
addition, teachers and students can jointly agree on appropriate learning strategies to follow, based on 
collaborative discussion around real factual data (Charleer et al., 2015). 


Table 2 presents an overview of the orchestration activities, actors, and pedagogical phases currently 
addressed by the analyzed case studies. Emphasis has been placed on supporting the orchestration 
activities of awareness and assessment and in the inter-active and post-active phases of the learning 
activities (rows 2 and 3). By contrast, other cells are empty or populated just by 1-2 exemplars. The 
empty cells in the table mean that the analyzed cases have not addressed certain orchestration 
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activities, targeted some actors, or provided support in some pedagogical phases. For example, the 
orchestration aspect that refers to adaptation, flexibility, and intervention has barely been explored in 
the analyzed cases. Thus, there is potential innovation in developing solutions that, for example, can 
perform automatic or semi-automatic interventions in student activities. There may be still under¬ 
attended actors as well. For example, providing LA tools to enhance student awareness or other 
orchestration activities in the physical classroom has not been deeply explored. Table 1 (columns 9 and 
10) also shows that there is potential to provide iterative support at a macro level. This can include 
providing continued LA support across sessions — bridging the physical world where interactive surfaces 
can capture some traces of f2f activity, with the digital remote access to resources. An alternative 
indicator of the maturity of this area of application is to observe to what extent the LA solutions can be 
readily deployed in authentic classrooms. Most of the examples analyzed describe lab-based scenarios, 
indicating that this area is rapidly growing but is still exploratory. The only examples of LA classroom 
tools mostly supported the orchestration aspect via the teacher's or public dashboards. 


Table 2: Maturity of Learning Analytics Applications Utilizing Interactive Surfaces. E=Educators, 
S=Students, R=Researchers, "=number of studies 


Orrhp^t ration artix/itip^ 

Pedagogical phases 


Pre-active 

Inter-active 

Post-active 

Adaptation 


E 


Management 


E 2 , S 2 

E 2 , S 2 

Assessment 

E 

R 2 , E 4 , S 3 

E, S 

Design 

e 2 ,r 


E 


Through the data gathered in the Learning Analytics for Redesign and LARAe.TT case studies, a large 
historical database regarding activities and achievements can be gathered. This can provide further 
benefit to teachers to evaluate, plan, and redesign their courses based on collective knowledge 
accumulated and curated by other teachers from the f2f learning experiences. As a result, the data 
captured in the micro-layer (e.g., in the classroom) can be valuable in the meso-layer (e.g., for course 
coordination and learning design of other subjects). By deploying these types of LA tools across different 
courses, this large LA knowledge database can provide a richer and more accurate, overarching view on 
courses and students. This can also help institutions improve the synergy between courses, redesigning 
educational strategies with a global vision. Alongside, historical data can help support students in 
planning their learning career, while personal progress and achievement information across courses and 
semesters can facilitate more accurate feedback. Deploying LA surface and tangible tools at a larger 
scale and performing longitudinal studies may, in the near future, help measure the impact of our 
approaches to support f2f learning in the meso-layer and eventually also in the macro-layer. 
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6 CONCLUSION 


This paper has presented a description of the orchestration activities, challenges, and pedagogical 
opportunities of applying LA solutions utilizing interactive surfaces to facilitate a range of f2f tasks. As 
illustrated in Tables 1 and 2, this area of research is still immature as the technology is co-evolving 
alongside pedagogical practices that are beginning to recognize the value that these pervasive devices 
may offer. Our analysis framework helped to characterize the design space in terms of orchestration 
activities that need to be addressed, along with the pedagogical phases that teachers or students need 
to accomplish in order to prepare for classroom sessions. This framework is promising to help 
decompose other LA deployments, especially for those scenarios that can be complex, involving iterative 
support across different classroom sessions, and considering different tools, and multiple sessions, LA 
target users, and orchestration activities. 

The paper points to the future work needed to support students directly, exploit further unexplored 
affordances of interactive surfaces (such as sketching), and also support other orchestration activities, 
such as adaptation, flexibility, intervention, management, design, and planning. Besides, most LA 
support through interactive surfaces has focused on providing visualizations and dashboards. Other 
analytics techniques look particularly promising for surface tools, given the activity data they are able to 
capture. These may include multi-modal analytics (e.g., traces of physical actions, or LA approaches for 
tasks that require handwriting and sketching using interactive surfaces), analytics from heterogeneous 
sources of data (e.g., coming from different devices or education software), and the provision of (semi) 
automated system interventions, alarms, or feedback. 
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